Skip to content

[SPARK-2340] Resolve event logging and History Server paths properly #1280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
andrewor14 wants to merge 7 commits into from
Closed

[SPARK-2340] Resolve event logging and History Server paths properly #1280

andrewor14 wants to merge 7 commits into from

Conversation

@andrewor14
Copy link
Contributor

@andrewor14 andrewor14 commented Jul 2, 2014

We resolve relative paths to the local file:/ system for --jars and --files in spark submit (#853). We should do the same for the history server.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 2, 2014

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 2, 2014

Merged build started.

@andrewor14 andrewor14 changed the title [WIP][SPARK-2340] Resolve History Server file paths properly [SPARK-2340] Resolve History Server file paths properly Jul 2, 2014
@AmplabJenkins
Copy link

AmplabJenkins commented Jul 2, 2014

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 2, 2014

Merged build started.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 2, 2014

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 2, 2014

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16289/

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 2, 2014

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 2, 2014

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16290/

@vanzin
Copy link
Contributor

vanzin commented Jul 2, 2014

LGMT. I'd have made the style changes in a separate PR, since they make the change appear much larger than it actually is.

@andrewor14 andrewor14 mentioned this pull request Jul 3, 2014
Closed
@AmplabJenkins
Copy link

AmplabJenkins commented Jul 3, 2014

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 3, 2014

Merged build started.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 3, 2014

Merged build finished.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 3, 2014

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16318/

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 3, 2014

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 3, 2014

Merged build started.

@andrewor14
Strip trailing "/" from logging directory
b393e17 
Before it was printing: Logging events to
file:/tmp/spark-events//pysparkshell-...
@AmplabJenkins
Copy link

AmplabJenkins commented Jul 3, 2014

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 3, 2014

Merged build started.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 4, 2014

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 4, 2014

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16323/

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 4, 2014

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

AmplabJenkins commented Jul 4, 2014

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16324/

@andrewor14
Copy link
Contributor Author

andrewor14 commented Jul 4, 2014

Tested on standalone and yarn clusters. This is ready to go from my side.

@andrewor14 andrewor14 changed the title [SPARK-2340] Resolve History Server file paths properly [SPARK-2340] Resolve event logging and History Server paths properly Jul 8, 2014
@andrewor14
Merge branch 'master' of github.com:apache/spark into hist-serv-fix
13ff406 
Conflicts:
	core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
@SparkQA
Copy link

SparkQA commented Jul 17, 2014

QA tests have started for PR 1280. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16784/consoleFull

@SparkQA
Copy link

SparkQA commented Jul 17, 2014

QA results for PR 1280:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16784/consoleFull

@pwendell
Copy link
Contributor

pwendell commented Jul 30, 2014

Jenkins, retest this please.

  • This patch

    FAILS

    unit tests.

@SparkQA
Copy link

SparkQA commented Jul 30, 2014

QA tests have started for PR 1280. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17463/consoleFull

@SparkQA
Copy link

SparkQA commented Jul 30, 2014

QA results for PR 1280:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17463/consoleFull

@asfgit asfgit closed this in a7c305b Jul 31, 2014
@andrewor14 andrewor14 deleted the hist-serv-fix branch August 2, 2014 02:27
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@andrewor14 @conviva-zz
[SPARK-2340] Resolve event logging and History Server paths properly
92c6ded 
We resolve relative paths to the local `file:/` system for `--jars` and `--files` in spark submit (apache#853). We should do the same for the history server.

Author: Andrew Or <[email protected]>

Closes apache#1280 from andrewor14/hist-serv-fix and squashes the following commits:

13ff406 [Andrew Or] Merge branch 'master' of github.com:apache/spark into hist-serv-fix
b393e17 [Andrew Or] Strip trailing "/" from logging directory
622a471 [Andrew Or] Fix test in EventLoggingListenerSuite
0e20f71 [Andrew Or] Shift responsibility of resolving paths up one level
b037c0c [Andrew Or] Use resolved paths for everything in history server
c7e36ee [Andrew Or] Resolve paths for event logging too
40e3933 [Andrew Or] Resolve history server file paths
wangyum pushed a commit that referenced this pull request May 26, 2023
@colinmjj
[Carmel 6640] Support create materialized view as datasource table (#…
f76533f 
…1280)
ashevchuk123 pushed a commit to mapr/spark that referenced this pull request Oct 27, 2025
@idzikovskyi
EZESC-2205: Fix the cookie issue for Spark MultiSchemeAuthHandler (ap…
b05af48 
…ache#1280)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@andrewor14 @AmplabJenkins @vanzin @SparkQA @pwendell